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Foreword 


On  18  December  1944  an  unforecasted  storm  hit  the  U.S. Navy’s  Third  Fleet 
off  the  Philippine  Islands.  When  the  storm  passed,  790  men  were  dead,  146 
aircraft  were  lost,  and  3  ships  were  sunk.  This  episode  demonstrates  that  ac¬ 
curate  wind  and  wave  forecasts  should  be  of  highest  priority  to  the  Navy  in 
both  war  and  peace. 

Since  only  good  models  can  give  good  forecasts,  this  report  compared  Navy 
models  to  both  satellite  observations  and  other  models  used  worldwide. 
Although  the  Navy  wave  model  worked  well,  the  wind  model  may  need 
improvement. 


W.  B.  Moseley 
Technical  Director 


A.  C.  Esau,  Captain,  USN 
Commanding  Officer 


Executive  Summary 


Analyses  from  the  U.S.  Navy’s  operational  wind  and  wave  models  were 
compared  to  similar  outputs  from  foreign  and  domestic  agencies.  Model  out¬ 
puts  were  ranked  by  agreement  with  satellite  wind  and  wave  observations  for 
two  days  (10  March  and  10  September  1986).  The  Navy’s  wave  model  ranked 
high  on  both  days,  but  the  Navy’s  wind  model  did  not. 
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Comparisons  of  Wind  and  Wave 
Models  with  GEOSAT:  Final  Report 


I.  Introduction 

Since  winds  and  waves  affect  naval  operations, 
models  to  analyze  (depict  present  conditions)  and 
forecast  (depict  future  conditions)  these  variables  are 
run  routinely  at  the  U.S.  Navy’s  Fleet  Numerical 
Oceanography  Center.  The  operational  wind  model  is 
described  by  Rosmond  (1981),  and  the  wave  model  is 
by  Clancy  et  al.  (1986). 

In  addition  to  this  Navy  effort,  other  domestic  and 
foreign  agencies  also  routinely  run  wind  and  wave 
models.  Are  their  products  any  better?  To  find  out, 
we  compared  the  Navy’s  model  analyses  with  those  of 
other  agencies. 

This  type  of  model  ranking  has  previously  been 
done.  Cavaleri  et  al.  (1982),  for  example,  compared 
wave  models  driven  with  identical  hypothetical  winds. 
More  recently,  Canada’s  Atmospheric  Environmen¬ 
tal  Service  ranked  several  wind  and  wave  models  (in¬ 
cluding  the  Navy’s)  using  moored  wind-wave  buoys 
for  verification  (Eid  et  al.,  1986;  Khandekar  et  al. 
1986;  Khandekar  et  al.,  1986). 

Conclusions  from  these  earlier  model  comparisons, 
however,  have  limited  applications  to  Navy  needs. 
Model  performance  in  hypothetical  winds  may  not 
relate  to  performance  in  actual  winds.  Also,  in  the 
Canadian  study,  the  reference  buoys  were  all  within 
200  nm  of  the  coast;  therefore,  the  rankings  applied 
to  a  nearshore  area. 

A  better  method  for  ranking  large-scale  wind  and 
wave  models  became  available  when  the  Navy’s 
GEOdesy  SATellite  (GEOSAT)  was  launched  in  1985. 
This  satellite,  like  the  earlier  GEOS-3  and  SEASAT 
satellites,  provides  real-time,  global  coverage  of  sur¬ 
face  wind  speeds  and  significant  wave  heights.  A 
description  of  its  orbit  and  other  properties,  as  well 
as  data  editing  and  matching  techniques,  are  described 
in  our  earlier  report  (Pickett  et  al.,  1986). 

For  this  report,  we  applied  our  previous  techniques 
to  rank  the  Navy’s  wind  and  wave  model  performance 
for  10  March  and  10  September  1986.  The  models  were 
ranked  against  those  of  Canada  (both  military  and 
civilian),  the  Netherlands,  the  Federal  Republic  of 
Germany,  Japan,  the  U.S.  Army,  the  National  Oceanic 
and  Atmospheric  Administration  (NOAA),  and  a 


private  U.S.  company  (Offshore  and  Coastal 
Technology).  Our  results  show  that  the  Navy  model 
rated  highly  on  both  days. 

II.  Method 

A.  Model  Output 

To  obtain  model  output,  we  requested  agencies  to 
send  us  copies  of  their  analyzed  (not  forecast)  surface 
wind  speed  and  significant  wave  height  fields  for  0000, 
0600,  1200,  1800,  and  2400  GMT  on  10  March  and 
10  September  1986.  Since  we  planned  to  compare  their 
routine  products,  we  did  not  ask  for  any  special  runs, 
grids,  or  levels. 

B.  Satellite  Data 

GEOSAT  data  are  available  within  a  few  hours  of 
being  observed.  They  are  transmitted  to  a  ground  sta¬ 
tion  as  the  satellite  passes  over  during  each  near-polar 
orbit.  On  board  the  satellite,  a  radar  altimeter  is  used 
to  provide  the  data  for  estimating  ocean  surface  winds 
and  wave  heights. 

Wind  speeds  are  derived  from  the  return  pulse 
magnitude,  since  the  pulse's  energy  is  absorbed  by 
wind-generated  capillary  waves  and  foam.  Significant 
wave  heights  are  estimated  from  the  leading-edge  slope 
of  the  return  pulse,  since  high  seas  spread  this  pulse 
and  reduce  its  slope.  Algorithms  to  convert  altimeter 
pulse  magnitudes  and  slopes  to  wind  speeds  and  wave 
heights  were  derived  and  improved  in  the  earlier 
SEASAT  and  GEOS  satellite  experiments.  More  details 
on  the  satellite  sensors  are  given  in  Kilgus  et  al.(1984). 

An  accuracy  study  of  GEOSAT  sensors  was  done 
by  Shuhy  et  al.  (1987).  They  examined  GEOSAT 
overflights  of  NOAA’s  moored  buoy  network  and 
computed  differences  in  simultaneous  wind  speed  and 
significant  wave  height  readings.  Their  root-mean- 
square  error  of  differences  between  buoy  wind  speeds 
(measured  10  m  above  the  surface)  and  GEOSAT  wind 
speeds  was  1.5  m/sec  for  276  pairs  of  observations. 
For  significant  wave  heights,  they  found  a  root-mean- 
square  error  of  0.3  m  for  332  observations.  These 
values,  which  are  surprisingly  small,  combine  both  the 
satellite  and  the  buoy  errors. 


C  Comparisons 

After  editing  the  two  days  of  GEOSAT  data  (to 
eliminate  transmission  errors,  nearshore  regions, 
islands,  and  ice),  we  averaged  wind  speeds  and  wave 
heights  over  1-sec  intervals  (1024)  values).  This  calcula¬ 
tion  gave  us  space  averages  of  approximately  7  km  by 
7  km. 

Next,  we  matched  GEOSAT  times  and  locations  to 
model  outputs.  Ideally,  we  would  have  preferred  all 
model  results  to  be  in  the  same  format  on  the  same 
grid.  Then  we  could  have  compared  the  same  points 
for  each.  However,  since  we  were  working  with  opera¬ 
tional  products  tailored  to  user  needs,  we  had  to  resort 
to  samples  using  any  points  that  matched. 

To  pick  our  samples,  we  selected  GEOSAT  wind  and 
wave  data  that  were  within  1 .5  hours  of  model  verifica¬ 
tion  times.  If  we  were  provided  magnetic  tapes,  we 
computer  scanned  both  the  model  values  and  the  time- 
matched  GEOSAT  values  to  extract  all  pairs  that  were 
within  50  km  of  each  other.  If  we  were  provided  grid 
maps  or  contour  charts,  the  process  was  more  subjec¬ 
tive.  We  plotted  the  time-matched  GEOSAT  data  on 
top  of  the  charts,  and  read  off  overlapping  points. 

Matching  the  wind  fields  presented  another  problem. 
As  satellite-derived  winds  are  from  such  a  different 
technology,  we  did  not  attempt  any  corrections  (height, 
stability,  etc.)  of  GEOSAT  or  model  surface  wind 
fields.  We  simply  compared  the  model  surface  winds 
as  we  received  them  to  GEOSAT  winds. 

Because  most  of  the  model  results  were  for  the  North 
Atlantic,  we  tested  the  global  models  in  two  ways.  First, 
they  were  compared  to  each  other  over  the  entire  world. 
Next,  North  Atlantic  subsets  were  extracted  and  com¬ 
pared  against  North  Atlantic  models. 

III.  Results 

A.  Statistics 

A  summary  of  the  wave  model  results  is  shown  in 
Table  1  (units:  meters),  and  a  summary  of  the  wind 
model  results  is  shown  in  Table  2  (units:  meters  per 
second).  Each  entry  covers  two  rows  and  lists  the 
agency  or  country,  the  region  (global  or  name  of 
region),  the  day,  the  mean  difference  of  the  matched 
points  (model  minus  GEOSAT),  standard  deviation  of 
these  differences,  root-mean-square  error,  scatter  in¬ 
dex  of  agreement,  and  number  of  matched  points  (see 
Panofsky  and  Brier,  1965,  and  Willmott,  1981  for 
definitions). 

B.  Variability 

Statistics  in  the  tables  are  rounded  to  one  decimal 
place  to  mask  sampling  variability.  We  established  that 
the  second  decimal  place  was  not  significant  by  running 
two  kinds  of  tests. 

First,  we  ran  the  Navy  wind  and  wave  models  against 
GEOSAT  for  four  other  days  to  estimate  day-to-day 


variability.  Next,  we  repeated  the  table  calculations 
with  a  variety  of  space  and  time  matching  criteria.  Both 
tests  produced  changes  in  the  table  statistics  in  the 
second  decimal  place.  Hence,  by  rounding  to  one  place, 
we  attempted  to  show  only  significant  changes. 

C  Interpretation 

In  general,  a  low  mean  in  the  tables  is  desirable  and 
shows  that  a  model  is  unbiased.  A  low  standard  devia¬ 
tion  is  also  important,  and  indicates  that  model  highs 
and  lows  are  in  the  proper  place  and  are  of  the  proper 
magnitude.  Information  in  these  two  statistics  can  also 
be  combined  into  a  single  number:  the  root-mean- 
square  error.  This  value  should  also  be  small  in  a  good 
model. 

The  scatter  index  is  similar  to  the  standard  devia¬ 
tion.  Whereas  the  standard  deviation  gives  scatter 
about  the  mean,  the  scatter  index  gives  scatter  about 
a  regression  line  through  the  data.  Good  models  should 
have  small  scatter  indexes. 

Finally,  the  skill  score  and  index  of  agreement  are 
similar;  both  measure  overall  model-GEOSAT  agree¬ 
ment.  The  higher  the  value,  the  better  the  agreement. 
Generally,  skill  scores  are  lower  because  randomness 
is  removed  during  calculation. 

D.  Comparisons 

We  included  all  these  statistics  in  the  tables  to  allow 
comparisons  with  other  results.  For  example,  when 
Clancy  et  al.  (1986)  compared  the  Navy  wave  model 
to  NOAA  moored  buoys,  they  used  root-mean-square 
errors  and  scatter  indexes.  Their  root-mean-square 
errors  ranged  from  0.7  to  1.3  m.  In  Table  1,  our  Navy 
global  wave  model  root-mean-square  errors  are  similar 
(0.9  and  1.1  m).  Their  scatter  indexes  were  the  same 
as  ours  (0.3  to  0.4).  Apparently,  comparisons  with 
either  buoys  or  satellites  give  comparable  results. 

As  mentioned,  the  Navy  models  were  also  tested  in 
Canada.  These  coastal  North  Atlantic  tests  (Khandekar 
et  al.,  1986)  showed  a  root-mean-square  error  of  4.3 
m/sec  for  the  Navy  wind  model  (we  had  3.9  and  4.0 
m/sec)  and  2.0  m  for  the  wave  model  (we  had  0.6  and 
0.8  m).  We  found  smaller  errors  using  the  North  At¬ 
lantic  than  they  found  using  coastal  data.  This  differ¬ 
ence  is  probably  because  the  coarse  grid  of  the  Navy’s 
global  model  does  not  properly  depict  coastal  regions. 

E.  Rankings 

The  results  in  Tables  1  and  2  are  rearranged  in  Tables 
3  and  4.  Table  3  shows  the  10  March  1986  model  rank¬ 
ing  based  on  three  of  the  above  indexes.  Table  4  shows 
the  same  type  ranking  for  September  1986.  In  both 
tables,  models  are  grouped  as  regional  (or  a  subset) 
and  global. 

IV.  Conclusions 

Our  first  conclusion  is  that  the  Navy  wave  model 
appears  to  work  well.  In  spite  of  its  simpler  physics, 


compared  to  newer  models,  no  other  model  beat  it 
consistently.  Advantages  of  advanced  physics  were  not 
evident  in  our  tests. 

The  major  region  where  the  Navy  wave  model  had 
problems  was  in  the  southern  ocean.  Figure  1  shows 
that  most  wave  model  errors  occurred  off  Antarctica. 
They  probably  stemmed  from  poor  wind  input  or  poor 
ice  edge  locations  (which  gave  wrong  fetches). 

Our  second  conclusion  is  that  the  Navy  wind  model 
may  need  improvement.  As  mentioned  earlier,  rank¬ 
ing  wind  models  was  more  subjective  because  of  the 
uncertainty  associated  with  model  and  GEOSAT  levels. 
Nevertheless,  in  both  the  Canadian  study  and  in  this 
study,  the  Navy  wind  model  did  not  do  as  well  as 
others. 

Our  final  conclusion  is  that  GEOSAT  provides  an 
ideal  way  to  evaluate  wind  and  wave  models.  In  addi¬ 
tion,  its  data  could  be  injected  directly  into  models. 
This  combination  of  testing  to  find  the  best  model  and 
assimilating  GEOSAT  observations  into  that  model 
should  result  in  a  significant  leap  in  the  accuracy  of 
wind  and  wave  data  going  out  to  the  Fleet. 
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Figure  1.  World  chart  showing  areas  of  differences  (in  meters)  between  Navy  wave  model  and  GEOS  AT 
on  10  September  1986. 


Table  1.  Statistics  of  differences  between  wave  models  and  GEOSAT  for  two  days. 
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-  Compflrfxl  by  hand  kom  charts  F  -  Rurl  wiiti  US  NOAA  winds 

-  Compflffld  by  campufcK  from  magnetic  tap*  G-  Ftanwttfi  US.  Navy  winds 

-  InsuWptwnl  otiwfvsMona  lw  Skill  scorn  H  -  Nertt* «n  tWsp**r8  orty 

-  Subs*!  ol  global  mndri  I  -  hand  drawn  antfpia 

Run  on  dnshtop  CQFnputer 


Table  2.  Statistics  of  differences  between  wind  models  and  GEOSAT  for  two  days. 
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Table  3.  Model  ranking  based  on  GEOSAT  agreement  on  10  March  1986. 
Brackets  indicate  ties.  Models  with  less  than  25  observations  were  not  included. 


Rank  by 

Skill  Score 

Rank  by  Index 
of  Agreement 

Rank  by  Root 

Mean  Square  Error 

Regional  Wave  Models 

Netherlands 

U.S.  Navy 

Canadian 

U.S.  Navy 

U.S.  Army 

Canadian  2 

U.S.  Navy 

Fed.  Rep.  Germany 

Netherlands 

OCTI 

U.S.  Army 

Canadian  Military 

Fed.  Rep.  Germany 

U.S.  NOAA 

Canadian  1 

i 

OCTI 

Netherlands 
^Fed.  Rep.  Germany 

Canadian  Military 

U.S.  NOAA 

Canadian  1 

Canadian  2 

OCTI 

U.S.  Army 

U.S.  NOAA 

Canadian  Military 

Canadian  1 

Global  Wave  Models 

U.S.  Navy 

U.S.  Navy 

U.S.  Navy 

U.S.  NOAA 

U.S.  Army 

U.S.  NOAA 

U.S.  Army 

U.S.  NOAA 

U.S.  Army 

Regional  Wind  Models 

Fed.  Rep.  Germany 
(^Canadian  1 

Canadian  2 

U.S.  Navy 

/ 

Fed.  Rep.  Germany 

Netherlands 

Canadian  2 

V 

'"Canadian  1 

Fed.  Rep.  Germany 

Canadian  2 

U.S.  NOAA 
j  Canadian  1 

(u.S.  NOAA 

Netherlands 

U.S.  Navy 

U.S.  NOAA 

V 

[  Netherlands 

U.S.  Navy 

Global  Wind  Models 

U.S.  NOAA 

U.S.  NOAA 

U.S.  NOAA 

U.S.  Navy 

U.S.  Navy 

U.S.  Navy 
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Table  4.  Model  ranking  based  on  GEOSAT  agreement  for  10  Sept.  1986. 
Brackets  indicate  ties.  Models  with  less  than  25  observations  were  not  included. 


Rank  by  Rank  by  Index  Rank  by  Root 

Skill  Score  Agreement  Mean  Square  Error 


Regional  Wave  Models 


U.S.  Navy 
Netherlands 

V 

Fed.  Rep.  Germany 
U.S.  NOAA 
OCTI 

Canadian  Military 


U.S.  Navy 

Netherlands 

U.S.  NOAA 

Fed.  Rep.  Germany 

Canadian  Military 

OCTI 


Netherlands 

U.S.  Navy 

U.S.  NOAA 

Fed.  Rep.  Germany 

Canadian  Military 

OCTI 


Global  Wave  Models 


U.S.  Navy  U.S.  Navy  U.S.  Navy 

U.S.  NOAA  U.S.  NOAA  U.S.  NOAA 


Regional  Wind  Models 


U.S.  NOAA 

Netherlands 
Fed.  Rep.  Germany 
U.S.  Navy 


U.S.  NOAA 

Netherlands 
Fed.  Rep.  Germany 
U.S.  Navy 


U.S.  NOAA 

Netherlands 
Fed.  Rep.  Germany 
U.S.  Navy 


Global  Wind  Models 


U.S.  NOAA 
U.S.  Navy 


U.S.  NOAA 
U.S.  Navy 


U.S.  NOAA 
U.S.  Navy 
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